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ABSTRACT 

Studies showing that even though certain factual 
questions are directly related to information in passages they ate 
also easier to answer without reference to the passage are the basis 
for this study which attempted to define qualities of relationship 
which might Le responsible for the ease of ar.cwc rabiiit y. Three Tests 
of uisitic r of Information (TAX) composed of paragraphs and 

m u 1 1 i p le- c ho ic e items designed,- including a phantasy test 

(TAI^-P) , a lexical substitution tost (TAI--LS) in which a key word 
was replaced by an artificial word, and a semantic anomaly test 
(TAI--SA) in which relationships were distorted by manipulations of 
parts of speech. Reliabilities were obtained by using a sample of 
approximately 75C j unior- h igh-schooi students. In addition, the 
experimental tests and a reading subtest of the Sequential Tests of 
Educational Progress (STEP) were administered to 240 
junior-high-schcol students, half of whom read the paragraphs ai.d 
answered the questions and the other half answered only the 
questions. IteiTs judged for passage related ness were analyzed using 
two-way analysis of variance procedures. Results indicated that 
TAI--P, STEP, and TAI--LS items did not differ significantly in 
either the passage or nonpassage conditions, while the TAI--SA test 
items did. References, tables, and sample test items are included. 
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PASSAGE-RELATED READING COMPREHENSION QUESTIONS 
J. Jaap Tuinman 

Reading Program Institute for Child Study 

Indiana University 

When questions about a reading passage can be answered by a subject 
who has not read the passage, doubt may arise about the validity of 
those questions as estimators of how well that student can read, 

Preston (1964), Weaver and Bickley (1967), Bickley, Weaver and 
Ford (1968), Weaver, Bickley and Ford (1969) and Mitchell (1967) have 
shown that a large number of test items in commercial reading tests is 
characterized by very low passage-relatedness. That is, the expected 
mean score for subjects taking those tests without the reading passages 
present exceeds considerably the mean score expected on the basis of 
chance only* 

A question of interest is, which item characteristics control 
the degree of item answerability under the passage-out condition, i.e. 
its passage-relatedness. Weaver c.s. contend that the more factual 
items are less passage related. These items were in their studies 
relatively easy even under the passage-out condition. Since factual 
items generally are thought of as those items which are directly 
answerable from the text, something of a paradox arises if the above 
contention is true: the more passage-related items (by one criterion) 

are the less passage-related items (by another criterion). 

In the present study the writer explored this paradox, referred 
to as leaver's paradox", somewhat further. In the first phase of the 
study three experimental tests containing 40 four foil multiple choice 

Paper to be read at the Annual Meeting of the AERA, New York, February, 1971 
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items each, were developed, (Appendix) The tests differed in the format 
of the passages on which the items were based. The Test of Acquisition 
of Inf ormation--Phantasy (TAI--P) contained phantasy stories of a 
science-fiction type, written by the author. The TAI- -Lexical 
Substitutions (TAI--LS) consisted of passages in which a key word 
governing the interpretation of the passage was removed and replaced 
by an artificial word. The TAI--Semantic Anomaly (TAI--SA) contained 
passages in which normal relationships among "events" had been distorted 
by manipulation of parts of speech. The passages kept their syntactic 
integrity, however. After repeated tryouts, involving a sample of 
approx, 750 junior high school students, relatively high reliabilities 
were obtained. (KR-20's between ,82 and .87), 

In the second phase of the study these tests were administered, 
together with a number of factor markers, to a sample of 240 7th, 

8th, and 9th graders. In addition, a subset of 40 items from the 
STEP reading test, form 3A, was administered. This repeated measure 
design allowed an in depth analysis of the relationships among the 
four reading measures, employing factor analysis. The results of this 
part of the study will be reported elsewhere. (Tuinman, 1970; Tuinman, 
Weaver, and Dekle, 1971). 

Prior to the second phase test administration, the items from the 
experimental tests and from the STEP had been rated by five judges on 
a four point scale indicating the degree to which each item could be 
answered directly from the passage. (Mean reliability of the summed 
ratings was .77). On basis of these ratings each test could be divided 
into three subtests of 10 items, with exclusion of the 10 items closest 
to the mean of the rating scale. These subtests were labelled "item-type" 
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subtests. The issue of passage-relatedness of the items was 
studied by sampling from the entire body of data in such a fashion that 
a 2 x 3 complete factorial design resulted. Factor A, passage condition, 
had two levels (passage-in; passage out); Factor B, item-type, had as 
its three levels: highly directly answerable from the passage, moderately 

directly and highly indirectly. The results for each of the tests were 
analyzed seperately because of the difficulty of the raters to keep 
their interpretation of the concept "directly answerable" invariant 
across the different passage formats. The present paper reports on this 
phase of tic study. (See tables 1 and 2.) 



Insert tables 1 and 2 here 

The data were analyzed by means of two-way Anova f s and subsequent 
Scheffe Multiple Cotaparisons . 

The results did not substantiate Weaver's paradox unequivocally. 

The TAI--P and the STEP subset featured normal prose, matching that 
of the tests analyzed by Weaver et al. For these tests it was expected 
that the i-'ems directly derived from the passages would be easiest. 
However, no significant differences among these subtests were found 
under the no-passage condition. The same was true for the TAI-LS items. 

The results of the TA I- -Semantic Anomaly, on the other hand, suggest 
a modification of the contention of Weaver c.s. that factualness is the 
key word to explaining passage-relatedness. For this test, the items 
most directly answerable from the passage, were decidedly the most 
difficult under the passage-out condition. (p<,01) The following 
interpretation of these results is suggested. 
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When a subject is reading an item, a structure of expectancies Is gen- 
erated* Those expectancies are based, at least in part, upon the 
experienced relationship among "events" implied in the question. Nor- 
mal prose rewards such expectancies in the sense that its content 
largely coincides with them; this reduces the need of reading as a 
source of Information for the answering of those items* It aeeios 
reasonable to expect that the type of question which is normally 
relatively easy becomes the type of question which is relatively hard, 
when the criterion for correctness Is changed to the evtent th?t the 
answers generated by the normal structure of expectancies are wrong and 
vice versa. 

The TAI--SA contained many items which were highly factual, and which 
could be answered by means of a direct referral to the text* Yet, in the 
case of these tests, those items were very hard to answer when the 
passage was removed. They displayed the characteristics of the items 
called "inferential items” by Weaver et al, It seems that factual 
Items based on regular prose are easy under the passage-out condition, 
not because they refer to facts, nor because they were derived directly 
from the passage, but becausi such items generate a structure of ex- 
pectancies which are rewarded by the passage. This very function seems 
to interfere with the measurement of the actual control of the reading 
passage over the item-answering behavior of the readers. 

In onler to measure the extent to which a subject does and can 
read, in some situations items seeetn to be required which exclude all 
sources of information but the reading passage itself. In the course of the 
research reported here, passage formats have been developed 
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which drastically reduced the proportion of Items answered under the 
passage-out condition. (E.g., for the TAI--SA tie mean score was 
8.01, as opposed to 13.66 for the STEP Items, under the passage-out 
condition. Ecth tests had 40 items each and comparable means under the 
passage-ln condition.) The data treated In the present paper a L Low 
a further refined understanding of which cha rac terlst lcs determine 
answerability of Items based on prose passages. It Is believed that 
this understanding has practical Implications for the development of 
more measures of ihe acquisition of Information irom reading 

pS6»ages--a behavior which Is taken to he a part of the set of behaviors 
of which reading comprehension is comprised. (Carroll, 1968) 
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Mote " To homogenize variances a Jog (X + I) transformation was performed; the analyses re] 
here are on the raw scores since the transformation did not reverse any inferential 



Table 2 



Analysis of Variance for 
TAI-P; TAI-LS, TAI-SA and STEP-40 



Test 


Source 


af 


SS 


MS 


F 




Passage Condition 


1 


589*07 


589.07 


159.81** 


TAI-P 


Item Type 


2 


54*02 


27*01 


7.30** 




PC X IT 


2 


1*31 


*65 






Error 


234 


862*20 


3*69 





TAI-L3 


Passage Condition 
Item Type 
DC X IT 
Error 


1 

2 

2 

234 


432.02 
10.41 
♦ 16 
769.00 


432.02 

5.20 

.08 

3.29 


13 1 .46** 
1.58 
.02 




Passage Condition 


L 


703.84 


703.84 


190* 79** 


TAI-SA 


Item Type 


2 


42.41 


21.20 


5 . 75** 




PC X IT 


2 


41.93 


20*96 


5.68** 




Error 


234 


863.23 


3.70 







Passage Conditions 


1 


173.40 


173.40 


64.74** 


STEP-40 


Item Type 


2 


101.46 


50.73 


18.94** 




PC X IT 


2 


55*33 


28.16 


10.51** 




Error 


234 # 


626*80 


2.68 





*''< p ^ .01 
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APPENDIX 

Sample stories and questions from the three Experimental Tests 



Stories 



Test of Acgui3i t io n of Information : Phantasy (TAI-P) 

Though the Mogs are very afraid of tigers, the one animal they 
probably fear even more is the Hog-eating fishu. Fishu, of course, 
is the Mog word for fish, '.he fishu most certain 1y is a dangerous 
animal. It can best be compared to our man-eating shark, except that 
the fishu can last longer on land than any other fish known to us. 

In this respect he even outlasts the king mackerel. That is exactly 
why the Mogs are sc* afraid of him. Any enterprising fishu who is in 
the right mood can come after them even if they stay away from the 
water. Altogether, the existence of the Hahn in hoiue 

country is a very disquieting situation. 

Te s t of Acquisitio n of Information ; L exica 1 Substitution (TAI-LS) 

Yesterday I saw a vurato. I was right on the spot when ft happened. 

Cars slowed down so the drivers could get a better look. People are 
like that, I suppose. Whenever something out of the ordinary happens, 
they r ve got to know what*s going on. Why this unexplainable urge? 

I don’t know. At any rate* a few seconds later 1 heard the sirens. 

The traffic came to a complete standstill. The street was entirely 
blocked by red emergency vehicLes. There were also a few white cars. 

They belonged to the policemen directing the traffic. I waited till 
the end. It took about three hours to battle the vurato. LuckiLy 
enough, the ambulance on the scene could be sent back empty- 

Test of Acquisition qlL Tn formations Se mflptjc Anomaly (TAl-SA) (l) 

Waves push up wind on seas and lakes and ponds. Everyone who 
has seen a wave has seen a sea. Sometimes the winds are hardly noticeable. 
But the waves may push up winds as tall as a whale standing on his tail. 

Test of Acquisition of Informat ion : Semant lc Anomaly (TAI-SA) (2) 

Trees want to g**ow new scientists to cut down the ones being replaced. 
Trees and numerous other plants also want to grow better scientists. So 
they make ways to find super scientists. 

Tiny redwood scientists growing near the ground have been cut from 
twigs. Treia found that these twigs could grow quickly into new scientists. 
Instead of taking almost 40 years to grow to a beginning scientist the 
new scientists may take only 60 short years now. 
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Questions 



Test of Acquisition of Information ; P hantasy (TAI-P) 

1* The Mogs are most ifraid of; (1) the man-eating shark, 

(2) tigers, (3) the fishu, (4) the king mackerel. 

2. One of the many animals the Mogs are afraid of ie the fishu 
Why in particular is this f ish dangerous? (1) it is very strong, 

(2) it is a member of the shark family, (3) it can stay alive on Land 
for a U-ng time, (4) its bite is poisonous. 

Test of Acqui si t ion of In forma t ion : Lexica 1 Subat i tut ion (TAI-LS) 

1. What is responsible for the fact that the c'.rs slowed down? 

(1) the drivers heard the siren, (2) the curiosity of the drivers, 

(3) the fact that the author was right there, (4) the road was blocked. 

2. What is a vurato? (1) a demonstration, (2) a fire, (3) a 
riot, (4) a parade. 

3. The author means by "chis Inexplainab Le urge" (in line A) j 
(1) man's desire to slow do.m, (2) man's desire to got a closer Inok^ 
(3) man's desire to do something extraordinary, (4) nan’s desire to 
satisfy his curiosity. 

4. The author left his office at l dclock. Thirty minutes leter 
he arrived at the place where the verato was. Right, after everything 
was under control he left for home. At what time did ho arrive at his 
home? (Ue lived 30 minutes from the vurato.) (1) 3 o'clock, (2) 

4 o’clock, (3) 5 o’clock, (4) 6 o’clock. 

Test of Acquisition of Information ; Semant i c A no m a 1 y (TAI-3A) 



1. According to the story, which cl the following is true? 

(1) waves are caused by winds, (2) winds are caused by waves, (3) 
the winds must push up waves, (4) winds are never noticeable. 

2. In the story what is being compared with a whale standing 

on his tail? (1) the Height of the tail, (2) lakes and seas and ponds, 

(3) the height of the waves, (4) the size of .“he winds. 

3. Which of the following stntemerts can not be correct? (E.ook 
back at the atory to check.) (1) John has seen a sea, (2) John has 
seen a wave and not a sea, (3) John has not seen a sea and a wave, 

(4) John has seen a wave, 

Story 



1. The new practice takes 60 instead of 40 yeara. The author 
sees this as a redwet'^n in time. This clear from the use of which 
word? (1) almost, (2) . ily t (3) instead, (4) now. 

2. The type of scientists talked about in this story finds Its 

origin in: (1) redwood experts, (2) super scientists, (3) die ground, 

'•1 • - — 



3. What happens to the scientists which are being replaced? 

(1) they get new positions, (2) they are made to grow, (3) they change 
Into something else, (4) they are being cut cV>m. 

4. Judging on the basis of the story, who hag a major interest in 
getting topnotch scientists? (1) the nation as a whole, (2) scientific 
scholars, (3) the universities, (4) the nation’s vegetat-lon, 




